Problem: EC2 Instance Accessing S3 Over the Public Internet

Currently, our EC2 instance is communicating with the S3 bucket over the public internet.

Why is this a problem?

• Security Risks: Traffic traveling over the internet is vulnerable to attacks, such as man-in-the-middle (MITM) attacks, where an attacker intercepts and potentially steals sensitive data.
• Exposure of Credentials: If attackers gain access to your AWS access keys, they could compromise your AWS account.
• Increased Latency: Internet-based communication adds unnecessary delays compared to private AWS network communication.

Solution: Use VPC Endpoints for Secure, Private Access

💡 VPC Endpoints allow secure, private connections between your VPC and AWS services like S3, without sending traffic over the public internet.

How VPC Endpoints Help:

✅ Traffic stays within AWS – no exposure to external threats.
✅ No need for an internet gateway, NAT gateway, or VPN.
✅ Lower latency and better performance compared to internet-based access.

🔹 Analogy: If an internet gateway is like a front door that connects your VPC to the public internet, then a VPC endpoint is like a private door that directly connects your VPC to AWS services without leaving AWS's secure internal network.

🚪 Securely Access S3 Using a VPC Endpoint and Bucket Policies

This guide will walk you through setting up a VPC Endpoint for S3 and securing access with a bucket policy to ensure that only your VPC resources can communicate with your S3 bucket without using the public internet.

🛠 Step 1: Set Up Your Architecture

Before we begin, let’s establish the AWS components required:

1️⃣ AWS Components Overview

1. VPC (Virtual Private Cloud) – Your private AWS network.
2. EC2 Instance – A virtual machine inside your VPC.
3. S3 Bucket – Cloud storage for your data.
4. VPC Endpoint – A private connection between your VPC and S3 (avoiding the public internet).

2️⃣ Create a VPC

A Virtual Private Cloud (VPC) is your isolated network in AWS.

If you don’t have a VPC already, create one:

1. Go to AWS Console → VPC → Your VPCs.
2. Click Create VPC.
3. VPC name: awsops-vpc.
4. IPv4 CIDR block: 10.0.0.0/16 (a large private network).
5. Leave everything else as default and click Create VPC.

3️⃣ Create a Subnet

A subnet is a subdivision of a VPC where resources like EC2 instances are launched.

1. Go to VPC Console → Subnets.
2. Click Create subnet.
3. VPC selection: Choose awsops-vpc.
4. Subnet name: awsops-public-subnet.
5. CIDR block: 10.0.1.0/24 (smaller range inside the VPC).
6. Select an Availability Zone.
7. Click Create subnet.

**4️⃣ Attach an Internet Gateway

**

An Internet Gateway (IGW) allows your VPC to communicate with the internet.

1. Go to VPC Console → Internet Gateways.
2. Click Create internet gateway.
3. Name it awsops-igw.
4. Click Create internet gateway.
5. Attach it to your VPC:
   • Select awsops-igw, then Actions → Attach to VPC.
   • Choose awsops-vpc and attach.

5️⃣ Modify Route Table for Internet Access

A route table directs traffic within a VPC.

1. 1. In VPC Console → Route Tables → Create route table.
   2. Name: awsops-public-route-table.
   3. VPC: Select awsops-vpc.
   4. Click Create.
   5. Attach Subnet:
      • Select awsops-public-subnet.
   6. Add Route:
      • Destination: 0.0.0.0/0 (internet traffic).
      • Target: Select awsops-igw.
   7. Click Save.
      
      

**🔑 Step 2: Launch and Connect to Your EC2 Instance

**

1️⃣ Launch an EC2 Instance

1. Go to AWS Console → EC2 → Instances.
2. Click Launch Instance.
3. Name: awsops-ec2-instance.
4. AMI: Choose Amazon Linux 2023 (or Ubuntu if preferred).
5. Instance type: t2.micro (free tier eligible).
6. Key pair: Create or select a key pair.
   • Click Create new key pair (if you don’t have one).
   • Download and save the .pem file.
7. Network settings:
   • VPC: Select awsops-vpc.
   • Subnet: Select awsops-public-subnet.
   • Auto-assign Public IP: Enabled.
   • Security Group:
     • Allow SSH (22) from your IP.
     • Allow HTTP (80) (if testing web apps).
8. Click Launch instance.

**2️⃣ Connect to Your EC2 Instance

**

1. In AWS Console → EC2 Console → Instances, select your instance.

2. Click Connect.

3. Use Session Manager if enabled, or SSH:

   ssh -i your-key.pem ec2-user@your-instance-ip

4. Verify AWS CLI is installed:

   aws --version

🔐 Step 3: Create Access Keys (Not Best Practice, but for Learning)

💡 Why? Access keys allow the EC2 instance to authenticate with AWS. Normally, we’d use IAM Roles, but this method helps understand authentication.

💡 What are credentials?
Credentials in AWS are like keys that let you access and manage AWS services securely. Without credentials, you won't have the permission to do things like viewing your S3 bucket list.

💡 Do I have credentials?
When you log into your AWS Management Console, you're already authenticated using your AWS user's details.

Running commands from an EC2 instance is a different story! You can think of your EC2 instance as another user that needs its own username and password (i.e. credentials) to get access to your AWS account.

Your account by default doesn't have credentials set up for running commands from an EC2 instance. You'll need to manually set these up to securely access and manage AWS services from your instance.

• Run aws configure

• The terminal is now asking us for an Access Key ID!

💡 What is an access key ID? Do I have one?
An access key ID is a part of a credential!

Your credentials are made up of a username and password; think of the access key ID as the username.

You don't automatically have one, but you can create access keys IDs through AWS IAM.

💡 What's the difference between access keys and key pairs?
You might remember key pairs from launching EC2 instances!

• Key pairs are used specifically for logging into your EC2 instances through SSH.

• Access keys, which are what we're learning about now, are credentials for your applications and other servers to log into AWS and talk to your AWS services/resources.

1️⃣ Create an IAM User and Access Key

1. Go to AWS Console → IAM → Users.
   

2. Click Add user.

3. Name it awsops-user.

4. 1. Click Security Credentials → Create Access Key.
      
   2. Select Command Line Interface (CLI) and check the acknowledgment box.
      

   Yellow Warning Banner Explanation:

   • Best Practice: Instead of using access keys, it’s recommended to attach an IAM role to your EC2 instance.
   • Why? IAM roles provide inherited permissions, making it easy to manage access securely without manually handling credentials.

   Access Key Use Cases:

   1. Local Code → Used when developing applications on your local machine that need AWS access.
   2. Application on AWS Compute → Used when an app runs on EC2, Lambda, or other AWS services and needs AWS access.
   3. Third-Party Service → Used when an external company (e.g., CrowdStrike) needs access to your AWS environment.
   4. Application Outside AWS → Used when integrating a non-AWS-hosted application with AWS services.

   Key Takeaway: IAM roles are generally more secure than access keys, but access keys are needed for specific use cases. 🚀

   • Select Next.

   • For the Description tag value, we'll write
     Access key created to access an S3 bucket from an EC2 Instance.
     

     5.  Click Create Access Key.
   

     6. Download the .csv file containing your Access Key ID and Secret Access Key (this is your only chance to download it).
   
   Downloaded .csv file:

5. 

   💡 What is the secret access key?
   The secret access key is like the password that pairs with your access key ID (your username). You need both to access AWS services.

   Secret is a key word here - anyone who has it can access your AWS account, so we need to keep this away from anyone else!

   🚨 Best Practice: Delete the access key when the session ends and use IAM roles for long-term security.

**2️⃣ Configure AWS CLI on EC2

**

On your EC2 instance, run:

aws configure

Enter:

• Access Key ID from IAM.
• Secret Access Key from IAM.
• Default region (e.g., us-east-1).
• Output format (leave blank or use json).
  

🪣 Step 4: Connect to Your S3 Bucket

**1️⃣ Create an S3 Bucket

**

1. Go to AWS Console → S3 → Create bucket.

2. Bucket name: awsops-bucket-yourname.

3. Select a region.

4. Block all public access: ✅ (recommended).

5. Click Create bucket.
   
   

   Upload Files to the S3 Bucket

   1. Select your bucket → Upload.
   2. Click Add Files → Select any two files from your local computer.
   3. Click Upload

2️⃣ Verify S3 Access from EC2

Run:

aws s3 ls

List files inside your bucket:

aws s3 ls s3://awsops-bucket-yourname

💡 Problem: Your EC2 instance is accessing S3 through the public internet! Let’s fix that.

**🚪 Step 5: Create a VPC Endpoint for S3

1️⃣ Set Up a Private S3 Endpoint

1. Go to AWS Console → VPC → Endpoints.
   
2. Click Create Endpoint.
   
3. Service category: AWS Services.
   
4. Service name: Search for S3 (com.amazonaws.[region].s3).
   
   • Select the filter result that just ends with s3.
5. VPC: Select awsops-vpc.
6. Route Table Association: Select your main route table.
7. Policy: Choose Full Access (for now).
8. Click Create Endpoint.
   

💡 What's does Gateway mean?
A Gateway is a type of endpoint used specifically for Amazon S3 and DynamoDB (DynamoDB is an AWS database service).

Gateways work by simply adding a route to your VPC route table that directs traffic bound for S3 or DynamoDB to head straight for the Gateway instead of the internet.

Extra for Experts: As AWS evolved and more companies started using it for diverse and complex tasks, there was a need for more detailed control over how data moves within AWS networks. Gateway endpoints they could manage traffic but didn’t offer detailed settings - this led to AWS creating Interface endpoints, which offer more security settings.

2️⃣ Test the Endpoint [Private Access to S3]

Run:

aws s3 ls s3://awsops-bucket-yourname

✅ Success! Now traffic flows privately through the VPC instead of the internet.

🔒 Step 6: Secure Your S3 Bucket with a Policy

All attempts to access to S3 bucket will be completely blocked except the connections only being received from our newly created VPC endpoint.

1️⃣ Modify the S3 Bucket Policy

💡 What's a bucket policy?
A bucket policy is a type of IAM policy designed for setting access permissions to an S3 bucket. Using bucket policies, you get to decide who can access the bucket and what actions they can perform with it.

1. Go to AWS Console → S3 → Your Bucket.

2. Click Permissions → Bucket Policy.
   

3. Click Edit Bucket Policy and paste: (replace vpc-xxxxxxxx with your VPC ID):

   {
     "Version": "2012-10-17",
     "Statement": [
       {
         "Effect": "Deny",
         "Principal": "*",
         "Action": "s3:*",
         "Resource": [
           "arn:aws:s3:::awsops-vpc-project-yourname",
           "arn:aws:s3:::awsops-vpc-project-yourname/*"
         ],
         "Condition": {
           "StringNotEquals": {
             "aws:SourceVpc": "vpc-xxxxxxxx"
           }
         }
       }
     ]
   }
   

   {
     "Version": "2012-10-17",
     "Statement": [
       {
         "Effect": "Deny",
         "Principal": "*",
         "Action": "s3:*",
         "Resource": [
           "arn:aws:s3:::awsops-vpc-project-yourname",
           "arn:aws:s3:::awsops-vpc-project-yourname/*"
         ],
         "Condition": {
           "StringNotEquals": {
             "aws:SourceVpc": "vpc-xxxxxxxx"
           }
         }
       }
     ]
   }
   

   💡 What does this policy do?
   This policy denies all actions (s3:*) on your S3 bucket and its objects to everyone (Principal: "*")... unless the access is from the VPC endpoint with the ID defined in aws:sourceVpce.

   In other words, only traffic coming from your VPC endpoint can get any access to your S3 bucket!

   

   Don't forget to replace:

   BOTH instances of arn:aws:s3:::your-bucket-name with your actual bucket ARN.

   Handy tip: you can find your Bucket ARN right above the Policy window

   
   

   • vpce-xxxxxxx with your VPC endpoint's ID.

     • To find this ID, you'll have to switch back to your Endpoints tab and copy your endpoint's ID. Make sure it starts with vpce-
       
       
       

   • After replacing the default values with your resources' ARN or ID, double check that you haven't deleted the quote marks around any of your statements. Also check that you still have /* at the second Resource line!
     

   🚨 Ran into this error?
   This means you haven't replaced all the placeholder values with your resources' ARN or ID yet!

4. Save the policy.

5. Once we save the changes everything on the screen will populate red disclaimer boxes.

This has rightfully occurred as a result of policy blocking all the traffic from any resources that is not the VPC endpoint.

🔄 Step #7: Access Your S3 Bucket Again!

1. From your EC2 instance:

aws s3 ls s3://awsops-vpc-project-yourname

WHY IS THE CONNECTION BEING DENIED EVEN WHEN WE HAVE SET UP TO ALLOW CONNECTION FROM EC2 ?

• Select Subnets from the left hand navigation panel.

• Select your public subnet, which starts with vpc-subnet.

• Select the Route table tab.

The traffic is being routed via internet gateway from the EC2 instance instead of it routing through the VPC endpoint.

🔄 Redirect Route Table Traffic to the VPC Endpoint for S3

By default, traffic to S3 might be routed via the internet gateway (IGW), which causes public internet access and potential access denials. To fix this, we need to update the route table to ensure that S3 traffic is routed through the VPC Endpoint instead of the internet gateway.

🚀 Step-by-Step Guide to Redirect S3 Traffic to the VPC Endpoint

1️⃣ Verify the Existing Route Table Setup

1. Go to AWS Console → VPC.

2. In the left navigation pane, click Route Tables.

3. Find and select the route table associated with your awsops-public-subnet (or the subnet where your EC2 instance is located).

4. Click on the Routes tab.

5. Look for a route with:

   • Destination: 0.0.0.0/0
   • Target: igw-xxxxxxxx (Internet Gateway)

   If your S3 requests are getting access denied, it's because the traffic is being sent through this internet gateway instead of the VPC Endpoint.

2️⃣ Identify the VPC Endpoint Route Target

Before adding the correct route, we need to find the VPC Endpoint ID.

1. Go to AWS Console → VPC.
2. In the left navigation pane, click Endpoints.
3. Locate the VPC Endpoint for S3 (should be named something like com.amazonaws.[region].s3).
4. Copy the VPC Endpoint ID (something like vpce-xxxxxxxxxxxxxxxxx).

3️⃣ Update the Route Table to Use the VPC Endpoint

Now, let’s modify the route table to direct S3 traffic through the VPC Endpoint instead of the internet gateway.

1. Go to AWS Console → VPC → Route Tables.
2. Select the route table associated with your VPC and EC2 subnet (awsops-public-route-table).
3. Click on the Routes tab.
4. Click Edit Routes.
5. Click Add Route.
6. Set the Destination as pl-xxxxxxxxxxxx (Prefix List ID for S3 in your region).
   • To get the correct S3 prefix list ID:
     • Click VPC Endpoints in the left navigation pane.
     • Find your S3 VPC Endpoint and check the Prefix List ID (e.g., pl-68a5400a).
     • Use this ID as the route destination.
7. In the Target field, select your VPC Endpoint ID (vpce-xxxxxxxxxxxxxxxxx).
8. Click Save changes.

4️⃣ Validate That Traffic is Going Through the VPC Endpoint

To confirm that your EC2 instance is now using the VPC Endpoint instead of the internet gateway:

1. SSH into your EC2 instance:

   ssh -i your-key.pem ec2-user@your-instance-ip

2. Run the following command to access your S3 bucket:

   aws s3 ls s3://awsops-vpc-project-yourname

   ✅ If configured correctly, this should now work without errors.

3. Check whether traffic is going through the VPC Endpoint:

   curl -s http://169.254.169.254/latest/meta-data/network/interfaces/macs/

   This should list the network interfaces associated with your EC2 instance.

4. Look at the associated routes:

   curl -s http://169.254.169.254/latest/meta-data/network/interfaces/macs/<your-mac>/vpc-ipv4-cidr-blocks/

   Replace <your-mac> with the MAC address retrieved in the previous command.
   ✅ The output should now only include VPC CIDR blocks and exclude public IP addresses.

5️⃣ Test Access Without the VPC

To confirm that S3 cannot be accessed via the public internet:

1. Try running this command from a local machine (outside AWS):

   aws s3 ls s3://awsops-vpc-project-yourname

   ❌ This should fail with an access denied error, meaning public access is blocked.

.

🎯 Step #8: Access Your S3 Bucket (One Last Time)!

Final test to ensure only VPC traffic reaches the bucket.

1️⃣ Block Public Access

1. Go to S3 Console → Your Bucket → Permissions.
2. Enable Block All Public Access.
3. Save changes.

2️⃣ Validate Restricted Access

✅ EC2 Instance (inside VPC):

aws s3 ls s3://awsops-vpc-project-yourname

❌ Local Machine (outside VPC):

aws s3 ls s3://awsops-vpc-project-yourname

🚀 Success! Your S3 bucket is now fully secured with a VPC Endpoint and a restrictive bucket policy! 🎉